Data replication framework

ABSTRACT

Generally described, the present disclosure is directed to an eventually consistent replicated data store that uses, for its underlying storage, a computer software library that provides a high-performance embedded database for data. The replicated data store employs a plurality of hosts interconnected to one another, allowing for writes to any host and full awareness of membership across all hosts. With the data replication framework disclosed herein, various modes are allowed to be built up on top of the core system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/046,775, entitled DATA REPLICATION FRAMEWORK, and filed Oct. 4, 2013,now U.S. Pat. No. 9,734,199, which is a continuation of U.S. patentapplication Ser. No. 12/980,193, entitled DATA REPLICATION FRAMEWORK,and filed Dec. 28, 2010, now U.S. Pat. No. 8,554,762, the disclosures ofwhich are hereby incorporated by reference in their entireties.

BACKGROUND

Generally described, replication is a set of technologies for copyingand distributing data and database objects from one database to anotherand then synchronizing between databases to maintain consistency. Usingreplication, data may be distributed to different locations and toremote or mobile users over local and wide area networks, dial-upconnections, wireless connections, and publicly accessible networks ofnetworks, such as the Internet.

Transactional replication can be used to replicate transactional data,such as a database or other form of transactional storage structure.Database replication can be used to describe scenarios in which databasemanagement systems attempt to replicate data in order to ensureconsistency between redundant resources. Database replication cancommonly be associated with master/slave relationship between theoriginal and the copies. In a master/slave relationship, one databasemay be regarded as the authoritative source of data, and the slavedatabases are synchronized to it. The master logs the updates, and theupdates are then sent to the slaves in order to synchronize them. Theslave outputs a message stating that it has received the updatesuccessfully, thus allowing the sending (and potentially re-sendinguntil successfully applied) of subsequent updates.

Multi-master replication, where updates can be submitted to any databasenode, and are then sent through to other servers for synchronization, isoften desired, but may introduce substantially increased costs andcomplexity which may make it impractical in some situations. One commonchallenge that exists in multi-master replication is transactionalconflict prevention or resolution. Most synchronous replicationsolutions do conflict prevention. Conflict prevention is typicallyaccomplished by not considering a write operation completed until anacknowledgement is received by both the local and remote databases.Further writes wait until the previous write transaction is completedbefore proceeding. Most asynchronous solutions do conflict resolution.Conflict resolution is typically accomplished by considering a writeoperation completed as soon as the local database acknowledges thewrite. Remote databases are updated, but not necessarily at the sametime. For example, if a record is changed on two nodes simultaneously, asynchronous replication system would detect the conflict beforeconfirming the commit and would abort one of the transactions. Anasynchronous replication system would allow both transactions to commitand would run a conflict resolution during resynchronization. Theresolution of such a conflict may be based on a timestamp of thetransaction, on the hierarchy of the origin nodes or on more complexlogic.

Database replication becomes difficult when it the number of databasesand/or the locations between the databases increases. Typically, acentralized relational database may be used to store data for a varietyof services across several hosts. In such a system, a simple request fordata would be sent to all the hosts, and each of the hosts would need toaccess the relational database to obtain the requested data. Theplurality of access requests to the centralized relational database maystrain the database. One solution has been to use localized caches onthe hosts, to reduce the number of access requests to the centralizeddatabase. The localized caches typically store local copies offrequently accessed data, thereby reducing the number of access requeststo the centralized database. The use of caches may thus allow for somescalability. However, as the data requirements grow, and larger cachesare needed, there may be issues such as shortage of random-access-memory(RAM). The use of multiple caches may create coherency issues. Stickyrouting may not always be applicable to such systems. When the number ofhosts and associated caches is scaled to a large enough number, thecentralized relational database may simply get overloaded and becomeunresponsive.

One possible solution to the overloading of a centralized relationaldatabase has been to scale with partitions. Caches on the hosts may bepartitioned to access multiple relational databases. However, such asolution does not really improve availability, since the two databasesare not replicates of one another. Overall, basic caching is not idealsince cache parameters require tuning, partitioning becomes a necessity,use of more partitions means more failures and availability is notimproved.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a system diagram illustrative of a data replication frameworkincluding a plurality of hosts and local data stores;

FIGS. 2A-2D are system diagrams of the data replication framework ofFIG. 1 illustrating different modes of operation;

FIG. 3 is a block diagram of the data replication framework of FIG. 1functional modules;

FIG. 4 is a block diagram of the interconnection between some of themodules in FIG. 3;

FIG. 5 is a flow diagram illustrative of interconnection of hostsimplemented by the server registry;

FIG. 6 is a flow diagram illustrative of membership synchronizationacross hosts, as implemented by the membership module;

FIGS. 7A and 7B are flow diagrams illustrative of write and persist, anddetermination of member availability routines implemented by the datareplication algorithm;

FIG. 8 is a flow diagram illustrative of determination of querydistribution implemented by the query analysis module; and

FIG. 9 is a flow diagram illustrative of dynamic request routingimplemented by the dynamic request routing module.

DETAILED DESCRIPTION

Generally described, the present disclosure is directed to an eventuallyconsistent replicated data store that uses, for its underlying storage,a computer software library that provides a high-performance embeddeddatabase for data. The replicated data store employs a plurality ofhosts interconnected to one another, allowing for writes to any host andfull awareness of membership across all hosts. With the data replicationframework disclosed herein, various modes are allowed to be built up ontop of the core system. For example, a partitioning strategy may bebuilt on top of the system. An authoritative store may also be built ontop of the system. The underlying data replication framework storagedoes not need to change for new features to be added, and multiplelayers can be utilized at the same time.

Specifically, in one aspect, the data replication framework can maintaina network of interconnected hosts with updated membership stateinformation across the hosts without the use of a centralized storagesystem. In another aspect, the data replication framework can implementstorage agnostic data replication across the network of interconnectedhosts, where membership information is taken into consideration forreplication strategies. In another aspect, the data replicationframework enables consistent data replication across the network ofhosts using features such as version numbers of data records and hostsacting as redundant senders of data. In another aspect, access requestsfor data from the hosts can be dynamically routed to perform loadbalancing.

Although various aspects of the disclosure will be described with regardto illustrative examples and embodiments, one skilled in the art willappreciate that the disclosed embodiments and examples should not beconstrued as limiting.

The core replication system described herein runs on a cluster of hosts.The hosts are interconnected and each host is aware of the existence ofeach other host. The cache on each of the hosts is a complete copy ofthe data. Such a system provides for both better availability and betterscalability.

FIG. 1 is a system diagram illustrative of a data replication framework100. The framework includes a replicated storage product composed of thefollowing: a set of hosts 140A-140N each running a storage engine,170A-170N; replication systems 150A-150N running on each of the hosts140A-140N and client application programming interfaces (API) 160A-160N.The replication systems 150A-150N replicate data across the hosts,guaranteeing they converge to the same state, bootstrap new hosts andtransfer data when repartitioning. The client APIs 160A-160N can dolocal or remote reads of the data, coordinate writes/updates bycommunicating with the replication systems while providing severalconsistency levels, and understand partitioning and route requestsaccordingly.

Illustratively, the framework 100 supports relational queries from anyone of the hosts 140A-140N. Data is replicated on each of the hosts140A-140N on the local data store 170A-170N. Each of the local datastores 170A-170N may consist of a computer software library thatprovides a high-performance embedded database for data. As will bedescribed below, the plurality of hosts 140A-140N are interconnectedsuch that each host is aware of the existence of each other host. Insome embodiments, the data replication framework 100 may also be incommunication with an external relational data store 200 forauthoritative storage.

FIGS. 2A-2D are system diagrams of the data replication framework 100 ofFIG. 1 illustrating different modes of operation. FIG. 2A illustrates aread operation. FIG. 2B illustrates a write operation. FIG. 2Cillustrates a replication operation from one host to all other hosts.FIG. 2D illustrates operation of the framework while one host isunavailable. These modes of operation will be further described below.

As illustrated in FIG. 2A, when an access request for data is receivedat the data replication framework 100, the data is locally read at oneof the hosts 140A-140N. There is no need to for an access request fromthe hosts 140A-140N to a centralized relational database 200 for a dataread, since each of the hosts 140A-140N includes a local data store170A-170N including a replicate of all of the data that would be storedat the centralized relational database 200. In this framework, if thecentralized relational database 200 were to become unavailable, datawould still be available to be read at any one of the hosts 140A-140N.Additionally, the number of hosts may be scaled to any number,independent of the capacity of the relational database 200.

FIG. 2B illustrates a write operation. Data received by the datareplication framework 100 may be received at any one of the hosts140A-140N. Any write request going to a host in the cluster of hosts140A-140N will get written on the respective local data store 170A-170Nas well as sent out to all other hosts 140A-140N in the replicationsystem 100, without needing to replicate the data to the centralizeddatabase 200. FIG. 2B also illustrates the optional writing of data fromone of the hosts 140A-140N to a centralized location, which may be therelational database 200. The data written to one of the hosts 140A-140Nmay be written from the host to the centralized relational database 200.As an illustrative example, the data replication framework 100 mayreceive data indicating that the price of an article is $4.95. This datamay be received at host 140B, and locally stored in data store 170B.

FIG. 2C illustrates the replication of data from one of the hosts140A-140N to all other hosts 140A-140N. Data received from the datareplication framework 100 is written to one host from the plurality ofhosts 140A-140N. The data is then replicated from that host, acting assender, to all other hosts in the plurality of hosts 140A-140N, withoutthe use of the centralized database 200. Following on the example above,the article price of $4.95 received by host 140B would be replicated tohosts 140A and 140C-140N without writing the article price to thecentralized database 200.

Illustratively, persistent replication includes propagating data fromone host to other hosts and receiving acknowledgments from the otherhosts. When an acknowledgment is not received by one of the hosts, thesender retries to send the data until an acknowledgment is received.However, acknowledgments may be lost, and/or data changes may arrive tohosts out of order. The data replication from one host to other hostsimplemented by different embodiments of the data replication framework100 may be performed in conjunction with several features to improve theconsistency and availability of data across the hosts. Some examples ofthese features may include the use of version numbers associated withthe data being replicated, the use of recorders associated with thetransmission of data between hosts.

For illustrative purposes, let us continue with the example above, butassuming there are a total of 4 hosts, for simplicity of description.Host 140B receives data indicating that the price of a given article is$4.95. Host 140B then sends this article price to hosts 140A, 140C, and140D. Host 140B receives acknowledgments from hosts 140A and 140C, butnot host 140D. Therefore, host 140B retries to send the article price tohost 140D. However, assume that between the time of the send and theretry, host 140B receives data indicating that the price of the articleis now $1.49. Host 140B sends this data to hosts 140A, 140C, and 140D.Then, host 140B retries sending the price of $4.95 to host 140D, andthis time host 140D acknowledges receipt. Now, hosts 140A-140C have dataindicating $1.49 for the price of the article, while host 140D has$4.95. This creates an inconsistent group of hosts.

In some embodiments of the data replication framework 100, theinconsistency of data across hosts, such as for example the inconsistentarticle price above, may be addressed by the use of version numbers fordata records. Accordingly, in an illustrative embodiment, data recordsmay be given a version number which is incremented with each update. Ifa host 140A-140N receives a change of data for a record with an olderversion number than the version locally stored, then the host 140A-140Nwould reject the data record. In some embodiments, the version numbersfor the records may be provided by the centralized relational database200. The generation and use of version numbers is described in furtherdetail below.

Returning to the example above, this time with the use of versionnumbers, host 140B receives data indicating that the price of a givenarticle is $4.95, and this price would be associated with a version 1.Host 140B then sends this article price and version number to hosts140A, 140C and 140D. Host 140B receives acknowledgments from hosts 140Aand 140C, but not host 140D. Therefore, host 140B retries to send thearticle price with version number 1 to host 140D. However, assume againthat between the time of the first send and the second try, host 140Breceives data indicating that the price of the article is now $1.49,associated with a version number 2. Host 140B sends this price andversion number to hosts 140A, 140C, and 140D and receivesacknowledgments from hosts 140A, 140C, and 140D. Then, host 140B retriessending the price of $4.95 with version 1 to host 140D. Host 140D has alocally stored price of $1.49 with a version number 2 associated withit, and therefore host 140D rejects the price of $4.95 with versionnumber 1, and sends an acknowledgment to host 140B. Now, hosts 140A-140Dhave data indicating $1.49 for the price of the article. This createsconsistent data across the group of hosts.

FIG. 2D illustrates embodiments in which one of the hosts 140A-140N istemporarily unavailable to the other hosts for sending and/or receivingdata. When one of the hosts 140A-140N desires to replicate a data writeto all other hosts 140A-140N, as a sender, and one of the other hosts140A-140N is unavailable, there may be issues such as datainconsistencies and sender crashes that need to be addressed. Continuingwith the example above, the article price of $4.95 may be successfullyreplicated to hosts 140A and hosts 140D-140N, but not to host 140C,since host 140C is temporarily unavailable.

The data replication framework 100 may include functionalities such ascontinuous monitoring of the unavailable host. The continuous monitoringof the unavailable host helps to determine when data should be resent tothat host. The continuous monitoring of the host also helps to determinewhen to declare that host permanently unavailable. Details about theimplementation of the continuous monitoring are described further below.

In other embodiments, the host sending data, such as host 140B in theexample above, may become temporarily unavailable during sending, or atany other time, or the sender host's memory may reach capacity. Thesesituations may also cause inconsistent data replication across thehosts. In some embodiments, the data replication framework 100 mayinclude recorders. Recorders are hosts that are responsible forrecording data being sent from a host so that the recorders can laterprovide the data to the sending host if the sending host becomestemporarily unavailable. Recorders also receive the acknowledgments fromother hosts for the original sender. The use of recorders enablesconsistent replication across hosts 140A-140N.

Continuing with the example above, hosts 140A-140D had a price of $1.49with version number 2 on the local stores 170A-170D. Let us suppose thathost 140A then receives a price of $2.34 for the article, with a versionnumber 3. Host 140A writes this price of $2.34 to its local data store170A (see FIG. 2B). Host 140A will also want to replicate this data tohosts 140B-140D. In embodiments where at least one recorder is used,host 140A sends the data to hosts 140B-140D, and selects at least one ofthe hosts, say host 140B, as the recorder. Host 140A may receiveacknowledgments from host 140B and 140C, but not host 140D. Then, thehost 140A may become temporarily unavailable. At that point, host 140Bwould take over for 140A and resend the price of $2.34 to host 140D.Host 140B, as recorder, would also keep track of acknowledgmentsreceived from host 140D. Then, when host 140A becomes available again,Recorder 140B sends the acknowledgment information received from host140D to host 140A. Through the use of the recorder, the data changereceived by host 140A is successfully replicated to other hosts evenwhen host 140A becomes temporarily unavailable. Also, theacknowledgments are received by the host 140A, and therefore the changesare not attempted to be sent after the host 140A becomes available.

A recorder provides redundancy for the sender. Recorders are picked foreach change that will persist if the host 140A-140N that originallyreceived the change goes down, as well as to receive additionalacknowledgments from all hosts 140A-140N that they have received thechange. Recorders are described in further detail below in connectionwith the description of the data replication algorithm.

In yet another aspect, the data replication framework 100 can facilitatethe adding new hosts as group members without having to stop read and/orwrite operations on existing hosts and without losing data changesacross the existing hosts. A new host added subscribes to the group ofhosts 140A-140N. Data existing at the hosts 140A-140N is copied in onewrite operation from one of the peer hosts to the newly joining host,while the hosts 140A-140N continue to accept any new changes. A fullchange history of all changes made to the data, such as the severaldifferent versions of the price of an article in the example above, neednot be kept at any of the hosts 140A-140N.

To illustrate the addition of a new member (or host), let us continuewith the example above, where the group of existing hosts are hosts140A-140D. Each of hosts 140A-140D is aware of its respectivepeers/members. Host 140A has peers 140B, 140C, and 140D. host 140B haspeers 140A, 140C, and 140D. host 140C has peers 140A, 140B, and 140D.Host 140D has peers 140A, 140B, and 140C. Then, assume a new host 140Ejoins the group. Host 140E may be added to the list of peers for any oneof the existing hosts 140A-140D. Say it is added to the list of peersfor host 140A. The updated list of peers/membership is replicated fromhost 140A to hosts 140B, 140C, and 140D. Then, the data on host 140A iscloned onto host 140E. Host 140E sends an indication of readiness tohosts 140A-140D. The new changes arriving to hosts 140-140D arerecorded, but not yet sent to host 140E. These changes are sent afterhost 140E has been restored.

In still further embodiments, the data replication framework 100 canfacilitate consistency in scenarios where one of hosts 140B or 140Cbecomes temporarily unavailable before receiving the updated list ofpeers/membership from host 140A in the example above. The datareplication framework 100 enables membership and replication strategiesto be separate from one another, through the use of a server registryand a membership module, as well as a data replication algorithm, aswill be described in reference to FIGS. 3-7 below. When new hosts join agroup of existing hosts, there may be situations where races are createdand there may be risks of changes being lost. The changes acknowledgedfrom the newly joined host remain, and a purge message is sent whendurable, as a checkpoint. Then the list of peers/membership is passedwith the purge message. The use of purge messages is described infurther detail below.

Continuing the example above, assume host 140B becomes temporarilyunavailable. Host 140A sends a purge message of acknowledged changes formembership (of hosts 140A-140E) to hosts 140A, 140C-140E. Then host 140Bbecomes available again. When host 140A attempts to send a purge messageof acknowledged changes for membership (of hosts 140A-140E) to host140B, host 140B realizes that it needs to update its roster. Then host140B updates its roster adding host 140E to it, and sends anacknowledgment to host 140E. Then host 140A successfully sends the purgechange message to host 140B.

FIG. 3 is a block diagram of the data replication framework of FIG. 1.The data replication framework 100 enables the functionalities describedabove, as well as other features and functionalities to be describedfurther below through a server registry 304, a membership module 308, adata replication algorithm module 312, a query analysis module 316 and adynamic request routing module 320. Specifically, the data replicationframework 100 can include a server registry 304 and a membership module308 each corresponding to one or more server computing devices formonitoring, updating, and synchronizing membership information acrossthe hosts 140A-140N. The data replication framework 100 can furtherinclude a data replication algorithm module 312 corresponding to one ormore computing devices for replicating data across the hosts 140A-140N.The data replication framework 100 can further include a dynamic requestrouting module 320 and an associated query analysis module 316, eachcorresponding to one or more computing devices for rerouting datarequests among hosts 140A-140N in order to more evenly balance datarequests across the hosts 140A-140N.

One skilled in the relevant art will appreciate that the datareplication framework 100 can be associated with various additionalcomputing resources, such additional computing devices foradministration of content and resources and the like. Additionally,although the server registry 304, the membership module 308, the datareplication algorithm module 312, the query analysis module 316 and thedynamic request routing module 320 are logically connected to the hosts140A-140N, these modules may be geographically distributed throughout acommunication network in a manner to best serve various demographics ofhosts 140A-140N.

FIG. 4 is a block diagram of the interconnection between the serverregistry 304, the membership module 308, the data stores 170A-170N, anda core replication interface 424. With reference to FIG. 4, before a newhost can join a group of existing hosts 140A-140N, the new host must beable to access all existing hosts. Additionally, all existing hosts140A-140N must also be able to access each other. The server registry304 is a small non-persisted replication system which replicatesavailable hosts 140A-140N. This allows all hosts 140A-140N to be awareof each other and exchange information. When a host connection isestablished to the server registry 304, all existing host sessions140A-140N are instructed to connect to the new host. This forms a fullyconnected mesh network between all hosts 140A-140N; the hosts 140A-140Nare fully interconnected to one another. When a host connection is lost,it is removed from the server registry 304.

As illustrated in FIG. 4, the server registry 304 may include a memberstore interface 404 to exchange membership state in the member store 416of the membership module 308. The server registry 304 may also include areplicated data store interface 408 to remotely persist changes to thecore replication interface 424. The server registry may also include adirect replication interface 412 to remotely persist changes to aspecific member's data store 170A-170N. These interfaces 404, 408 and412 provide the core query and update methods. Illustratively, the corereplication interface 424 coordinates all replication interaction andaccesses the membership agent 420 of the membership module 308. Themembership agent 420 persists state via the member store 416, which inturn persists into the same data store as used by the data stores170A-170N.

In an illustrative embodiment, the member store 416, in communicationwith the membership agent 420, persists known group host members140A-140N and their state. The state of a host member 140A-140N may be,for example, eligible or ineligible. A host may be considered ineligiblewhen it is temporarily or permanently unavailable. The member store 416shares the same data store as that used for data storage on the hosts140A-140N. In other embodiments, the data store for the member store 416may be separate. A logical mapping may be used to persist member state,although it is indexed too. The member store 416 maps member ID tonetwork address and a state, where address may be an IP host name plus aTCP port. Member ID may be a randomly chosen ID. Additional fieldspersisted in the record may include the peer restoration parent, a lastmodified timestamp, and an optional timestamp indicating when a memberbecame incommunicado. Illustratively, a record exists for a given memberID, but multiple records may exist with the same network address. Thisallows data stores 170A-170N to be migrated to different hosts 140A-140Nand also allows dynamically assigned addresses to change. When a memberconnection to an address is confirmed, all other member records withthat same address are deleted from the member store 416. The membershipagent 420 is responsible for replicating member stores 416 in a way toensure that the stores 416 are ultimately consistent with each other.

When a member state indicates eligible, then the member is considered avalid member of the replication group. All changes are replicated to alleligible members. If a member is temporarily ineligible, changes will bereplicated to that member when it becomes eligible again. If a member isineligible for an extended period of time, the member will be marked aspermanently ineligible. All pending changes for that member can besafely discarded. A member state cannot transition backwards from beingpermanently ineligible to being eligible, because pending changes willbe permanently lost. Members marked as permanently unavailable mustacquire a new identity (a new ID) and then restore all data from aneligible peer. Once a permanently unavailable member record is createdin the member store 416, the only way for this record to be deleted isif the address is re-used for another member ID. A timestamp may bestored for each permanently unavailable member record. The permanentlyunavailable member records which exceed a maximum threshold of time,such as for example over a year, may be deleted. In various embodiments,the length of time may be configurable, and may be shorter or longerthan a year.

Another property stored in the record of the member store 416 is atimestamp indicating when a member started to be unavailable. The actualmember state is still eligible until the membership agent 420 declaresit to be permanently unavailable. Persisting the unavailable timestampallows the membership agent 420 to continue determining the availabilityof the member from where it left off, in case the member becomesavailable again. If the grace period has expired (for example, if themembership agent 420 was off for too long), the unavailable timestamp isreset to the current time.

The membership agent 420 gets notified when member sessions are createdand lost. The membership agent 420 is responsible for replicating themember store 416 between all the member sessions and it is responsiblefor declaring members as permanently unavailable. Because the membershiproster is fully replicated among all group members 140A-140N, a newmember only needs to connect to one seed member. This is how the datareplication framework 100 performs member discovery. The roster isreplicated to the new member, and the new member is replicated to allthe existing members. The new member does not need the seed anymore,because it has locally persisted the entire roster. Even though the datareplication framework 100 can be configured with multiple seed hosts,these hosts might disappear when they are released. The loss of seedhosts will not affect existing group members, but it will prevent newmembers from joining. The seed host set will need to be updated toinclude members which are currently available. Seeding via a centraldatabase lookup, or via multicast, or via a pluggable discoverymechanism are other possibilities in other embodiments.

FIG. 5 is a flow diagram illustrative of interconnection of hostsroutine implemented by the server registry. One skilled in the relevantart will appreciate that actions/steps outlined for routine 500 may beimplemented by one or many computing devices/components that areassociated with the server registry 304. Accordingly, routine 500 hasbeen logically associated as being generally performed by the serverregistry 304, and thus the following illustrative embodiments should notbe construed as limiting.

At block 504, when an eligible host connection is established to theserver registry 304, the server registry 304 receives the services ofthat host. At block 508, the server registry generates a hash table ofthe remotely addressable services on the hosts. The table may includereferences or identifiers for remote objects available to the hosts. Thetable may also include all other connection information necessary forthe hosts to access the remote objects. When a host connection becomesineligible, the server registry 304 removes the services of that hostfrom the table. At block 512, the server registry distributes the hashtable to all other eligible hosts. When a host connection isestablished, the server registry 304 instructs all existing hostsessions 140A-140N to connect to the new host. When a host connection islost or removed because of a host becoming ineligible, the serverregistry 304 instructs all eligible hosts to not connect to theineligible host. This forms a fully connected mesh network between allhosts 140A-140N where all the hosts 140A-140N are fully interconnectedto one another. The server registry 304 thus acts like a directoryservice without a central server; it floats among hosts themselves.

FIG. 6 is a flow diagram illustrative of a routine for membershipsynchronization across hosts implemented by the membership module. Oneskilled in the relevant art will appreciate that actions/steps outlinedfor routine 600 may be implemented by one or many computingdevices/components that are associated with the membership module 308.Accordingly, routine 600 has been logically associated as beinggenerally performed by the membership module 308, and thus the followingillustrative embodiments should not be construed as limiting.

As described in reference to FIGS. 4 and 5 above, host members 140A-140Nare uniquely identified and can be remotely accessed. All members areaware of the existence of their peers, and they routinely check toensure their availability. At block 604, the membership module 308listens for new member session connections. If a new member connects,the membership module ensures that membership information is persistedalong with the data replication framework's 100 data store, and anymembership changes are replicated to all other members 140A-140N atblock 608. In an illustrative embodiment, a new member need only connectto one host, or seed, to access the entire membership roster and becomean equal member of the group.

At block 612, the membership module 308 verifies whether a memberconnection has been lost. If it has not been lost, the membership modulecontinues to replicate the member store between member sessions (block608). However, if the member connection has been lost, at block 616, themembership module attempts a reconnect and/or verifies whether theremote member has attempted to reconnect. If the reconnect is successfulat block 620, the membership module continues to replicate the memberstore between member sessions (block 608). However, if the reconnect isunsuccessful, the membership module continues to attempt to reconnectand/or verify for attempts to reconnect from the remote host for athreshold amount of time at block 616. If within that threshold of time,the reconnection is successful, the membership module continues toreplicate the member store between member sessions (block 608). However,if the reconnect attempts are unsuccessful after the threshold amount oftime has been exceeded, the member may be declared permanentlyunavailable. This process is described further in connection with FIG.7B below.

When a member detects that another has been unavailable for too long, itcan declare it to be permanently unavailable Once a member has beendeclared permanently unavailable, any pending updates to it can besafely discarded. Should the permanently unavailable member attempt tojoin the group again, it is denied membership. The permanentlyunavailable member's data store is considered to be too stale and itneeds to generate a new identity and be restored from one of theavailable peers. Detection of temporarily and permanently unavailablemembers is described in further detail below.

FIGS. 7A and 7B are flow diagrams illustrative of the write and persistroutine of the data replication algorithm for available hosts, anddetermination of availability or permanent unavailability for hosts thatare temporarily unavailable. With reference to FIG. 7A, at block 704,data is written locally to one of the hosts 140A-140N. At block 708, thedata is persisted locally. At block 712, the data replication algorithmverifies whether a remote host to which data will be propagated isavailable. If the remote host is not available, the algorithm moves toblock 716, described in connection with FIG. 7B below. If the remotehost is available, the algorithm moves to block 720 to propagate thedata to the remote host. Block 712 of the write and persist routine maybe repeated for all remote hosts to which data is to be propagated forreplication. Block 720 of the data replication algorithm may be repeatedfor all hosts which are available. After the data has been replicated toall of the available remote hosts, the replication algorithm ends atblock 724. Block 716 may be repeated for all hosts which are unavailableon the first attempt.

As described briefly in connection with FIGS. 2A and 2B above, the datareplication framework 100 is designed to ensure that data is ultimatelyconsistent across all hosts 140A-140N. In some embodiments, the datareplication framework 100 may also support additional features tofurther improve consistency. Returning to FIG. 7A, changes replicated atblock 720 may be done by employing various replication strategies. Insome embodiments, an “at least once” strategy may be employed, where agiven change may be repeated more than once. In various embodiments, thechange may be replicated twice, three times, or more. In someembodiments, record version numbers may also be used to decide if anincoming change is stale and should be rejected.

In addition, in some embodiments, before or after verifying the remotehost is up at block 712, the data replication algorithm may designateone of the hosts as a recorder. The data replication framework 100 canbe thought of as having two types of message streams, each of whichdeliver changes from one peer host 140A-140N to another. The first typeof message stream may be a live stream, which includes changes createdon a group member (the change sender), for delivery to the availablegroup members. The recipients of these changes are either recording, ornon-recording, receivers. The second type of message stream may be astandby stream, which includes changes that were not delivered as a partof the live stream, and are delivered via the standby stream. The bulkof the standby changes occur on account of an unavailable member140A-140N missing changes that were delivered as a part of the livestream. Also, live stream messages that suffer from transmission issues,either in change delivery, or acknowledgment receipt, get resent as apart of the standby stream. Transmission of standby streams may beinitiated by the change sender, or the designated recorder.

Change operations may utilize record version numbers for reconcilingconflicts. If a change operation is received against a record versionnumber older than what exists in the data store 170A-170N, it can berejected. Several types of core operations may be performed on datarecords received during data replication, depending on designationsassociated with the data type, key, value, and version. For example, ina STORE operation, a record may be forcibly stored by deleting anyexisting record. The given record version number is assigned to thenewly stored record. In an UPDATE operation, a record may be inserted orupdated. If a record already exists, the given record version numbermust be larger than the existing record version. In a PURGE operation, arecord may be forcibly deleted by its key, ignoring any record versionnumber. In a DELETE operation, a record may be deleted by its key. If arecord already exists, the given record version number must be larger orequal to the existing record version. In a TRUNCATE operation, allrecords may be forcibly deleted for a given type.

Record version numbers may be compared using modulo arithmetic. This mayallow them to wrap around without affecting the comparison, providedthat the difference between the values is less than 2³². In someembodiments, timestamps may be used as version numbers. In someembodiments, record version numbers can be incremented with the help ofan external database. In such embodiments, timestamps need not be usedfor record version numbers.

The data replication framework 100 provides for direct replication topeers 140A-140N. All members within a replication group maintainconnections to each other. Remote calls can be made directly to anymember, even asynchronous calls. The membership agent 420 provides alist of eligible members and also notes which are unavailable. Directreplication is basically implemented by sending a change to eachavailable member, asynchronously. A change set can be sent from anymember, and this same member (the sender) is responsible for directlyreplicating the change to all peers in the group. It does this byissuing a simple remote method call. Since a peer can be unavailable,the sender is responsible for resending the change set when the peerbecomes available again. To guard against change loss, the senderpersists replication state, so that it can resume replication ofundelivered changes after restarting.

Because a sender can itself become unavailable all changes pendingreplication would be lost. When a replicated change set is created, thesender chooses additional peers to assume the role of recorders. Theamount of recorders required is defined by configuration. In someembodiments, two recorders may be used. When the primary sender receivesacknowledgment of a change, it also acknowledges to the recorders. If arecorder observes that the primary sender becomes unavailable, ithandles delivery of the changes of the primary sender.

Direct replication follows a commit and propagate strategy. However,because additional information is persisted pertaining the replicationitself in the data replication framework 100, the weaknesses of directreplication are avoided. In various embodiments of the data replicationframework 100, all replication state is persisted in the sametransaction as the change set (or change session), and so if replicationis not possible, the entire transaction reverts back to its previousstate.

As explained above, recorders are selected to redundantly storereplication state. More recorders improve redundancy, but may reduceoverall write throughput. The minimum and desired redundancy level maybe configurable. If the desired redundancy level is zero, then norecorders need be selected. Only the original sender is responsible forresending changes in a standby stream. If the operating system crashesbefore all changes have been delivered, then the sender might lose thechanges and not be capable to send the standby stream. As a result, thereplication group may be inconsistent.

Illustratively, all available hosts are recorder candidates, which areselected immediately before the changes are committed. Recorderselection distributes the load among the hosts. All members must acceptall changes. In some embodiments, random or round robin selection may beused for selection of recorders.

The sender of a change does not ordinarily receive its own changes back.Because the sender could lose its own non-committed changes after areboot, it needs to restore recent changes from its recorders. For thisreason, the recommended minimum redundancy level may be one in someembodiments. The sender is recorded as a receiver, but the change isinitially acknowledged. Upon start up, members request all of theirrespective acknowledged changes from all peers.

Change sets may be assigned a unique identifier, or ID, which isprovided by the initial sender. Change set IDs may be designed tosupport a natural ordering which loosely matches the order in whichchanges are sent. If a change could not be initially delivered, thestandby stream will attempt to match the original order.

In order to persist data, an extra replication state may be stored bythe sender of a change set, and the same state is stored by all selectedrecorders. The state is persisted in the same transaction as the changeset, eliminating edge cases caused by commit and propagate stylereplication. Two tables may store replication changes. The first tablemay map change set ID to a serialized change set, the sender ID, and anarray of recorder IDs. Sender and recorder IDs may be the same as memberIDs, and the order of recorders may determine the preference oftransmissions in the standby stream. Although a change set may include atime entry, a local timestamp may also be recorded in the replicationchange table.

The second table may contain a list of receivers for a change. It mayhave a composite key composed of the receiver ID and change set ID. Areceiver ID may be just a member ID. The mapped value is anacknowledgment, represented by the receiver's checkpoint counter value.The existence of this value indicates that the receiver has acknowledgedthe change, but it has not durably persisted it yet. The record isdeleted after it has been acknowledged and durably persisted.

After all receiver records for a change are deleted, the entrycontaining the serialized change set is deleted. In a healthyreplication group, where all members are available, the replicationstate data exists only for a short period of time. Replication state forunavailable members is kept until the member becomes available again oris declared permanently unavailable.

Senders and recorders do not persist self entries in the replicationreceiver table, but recorders do persist a pre-acknowledged entry forthe sender. This allows the sender to recover recently lost changeswithout requiring that changes always be broadcast back to it.

Following a peer data store transfer, the restored peer will have acomplete copy of the replication state from its peer. Informationpertaining to the peer may be removed after the transfer.

During initial propagation, immediately before a change set or changesession is committed, the complete set of replication group peers isexamined. From the replication group peers, recorders are selected basedon the desired redundancy level. Recorders are also persisted asreceivers, and all replication state entries are inserted asunacknowledged. Following selection of recorders, the transaction iscommitted, and then propagation can proceed. As an optimization,recorders can store a pre-acknowledged entry for the sender to avoid oneadditional round trip. For each receiver, an asynchronous remote call ismade. For ordinary receivers, the call contains the sender ID, thechange set ID, the recorder IDs, and the serialized change set data.Recorders are also passed all non-recorder receiver IDs. All receiverswill apply the changes, but recorders will also insert replication stateentries within the same transaction. After the receiver has committedthe changes, it asynchronously notifies the sender and all recorders bypassing back its own ID, the change set ID, and its checkpoint countervalue. This is then persisted in the corresponding entry in thereplication receiver table, indicating acknowledgment.

The data replication framework 100 waits for at least one recorder toreceive the change. This avoids a change being committed locally, andthen the sender becoming unavailable before having had a chance toreplicate the changes to remote hosts. In some embodiments, hosts mayhave the option of waiting for acknowledgments from any kind of hosts,not just recorders.

The standby stream is a background thread which ensures that allunacknowledged changes are delivered. Every few seconds (or a differentconfigurable time period), the replication receiver table is scanned inits natural order. This order is receiver ID, change set ID. Change setorder approximately matches the time when the change was sent. To avoidoverwhelming receivers during transmission of a standby stream, sendsare not sent asynchronously. The sends are instead sent using a batchedmode.

FIG. 7B illustrates the routine performed by the data replicationalgorithm after a host was unavailable on a first attempt at block 712of FIG. 7A. At block 728, the host is declared unavailable. When theserver registry 304 notifies the membership agent 420 that a membersession is unavailable, the membership agent 420 then decides if itshould attempt a reconnect. If the unavailable member exists in themember store 416 then the membership agent 420 considers the member tobe temporarily unavailable and performs a reconnect attempt and/orverifies if the remote host has attempted a reconnect at block 732. Ifthe attempt is successful, and the remote host is available (block 736),the remote host leaves the unavailable state at block 740, and the datais propagated to the host at block 744, and the data replicationalgorithm ends at block 764. However, if the reconnect attempt at block732 is unsuccessful, and the remote host is still not available (block736), then a subsequent reconnect is attempted at block 748.

The first reconnect/verification attempt at block 732 is performedimmediately, and each subsequent reconnect/verification at block 748 isdelayed. The first delay may be one second, and it may keep doubling upto, for example, two minutes. A member leaves the unavailable state (andreconnect attempts cease) when a reconnect succeeds or the unavailablemember establishes a connection itself (block 752).

If a member is unavailable without interruption for an extended periodof time (block 756), it is declared permanently unavailable at block760. Illustratively, 20 hours may be a default threshold for theextended period of time. A shorter threshold merely means that anunavailable member might be declared permanently unavailable too soon,and might be forced to restore from a peer when it becomes available. Ifthe data store is large, a full peer restore might take a long time, inwhich case it might be preferable to have missed updates played backinstead.

When a member is declared permanently unavailable at block 760, thisstate transition is immediately replicated to all other members in thegroup. This ensures that if a member recently declared permanentlyunavailable reconnects, it gets promptly notified of its permanentunavailability state, no matter what member it connects to.

Because a member must be unavailable for an uninterrupted period of timein order to be declared permanently unavailable, restarting themembership agent 420 would interfere and possibly cause the verificationof the member state to take longer. For this reason, the start time ofthe member's unavailable state may be persisted in the member store 416.

Returning to FIG. 7B, before the membership agent 420 attempts areconnect at blocks 732 and/or 748, the membership agent 420 updates thepersisted member state indicating that is the member is stillunavailable. At a minimum, this simply updates the last modified fieldof the record. It may also set the start time for when the unavailablestate was entered. When a member leaves the unavailable state at block740, the unavailable start time field is cleared.

If the persisted member record already has an unavailable start time, itbecomes the effective start time if the record was modified within thegrace period. The grace period is defined to be double the maximum retryinterval, which may be about four minutes in some embodiments.

When a member accepts a potential peer member session, it queries themember store 416 for the state of the peer member. If the member is newor known to be available, then both members synchronize their memberstore data. If the member has been declared permanently unavailable, itis treated as a compromised host, for example a zombie. With comprisedhosts, the member store synchronization is not symmetrical. The memberstates provided by the compromised host are ignored.

By ignoring state provided by comprised hosts cascading the memberunavailability state to connected members is prevented. In such ascenario, two members might declare the other permanently unavailable.This might be caused by a temporary network partition which lastedlonger than the detection threshold for permanent unavailability. Asimple cause might be a disconnected network cable, but not a poweroutage. In the worst case, an entire replication group could be declaredpermanently unavailable following a network partition. Since each sidethinks the other side is permanently unavailable, it does not replicateany member state. This leads to a permanent network partition. In thisstate, members in the replication group will never reach consistency.Also, changes made by one side of the partition will always be ignoredby the other.

Compromised hosts are most likely caused by a host being offline longenough to be declared permanently unavailable. The compromised hostaccepts the member states as provided by its peers, and upon doing so itlearns that it was declared permanently unavailable. It then creates anew member ID, rejoins the group, and performs a backup restore from apeer. Logic for detecting compromised hosts prevents the entirereplication group from being declared permanently unavailable followingan extended network partition. In some embodiments, the data replicationframework 100 can detect a permanent network partition, by simplyobserving pairs of compromised hosts, or two permanently unavailablemembers declaring the other to be permanently unavailable. The framework100 may log errors, repeatedly, to ensure detection by a log scanmonitor. It may be necessary in some embodiments to force all hosts onone partition to restore from the other.

Simple algorithms can be devised to automatically repair the permanentnetwork partition. In one embodiment, the partition with majority (moremembers) may be selected to be correct. The minority may restore frompeers in the majority.

In some embodiments, resiliency against extended network partition maybe achieved by increasing the threshold time for declaring a memberpermanently unavailable. With a threshold of 20 hours, a networkpartition which is resolved within 20 hours may cause no harm. If longernetwork partitions are expected, the threshold may be increased to alonger period of time. In other embodiments, one partition of hosts maybe deactivated during a large scale network outage, and only reactivatedonce the network is restored. In such embodiments, only the reactivatedhosts will be declared permanently unavailable, and they can restorefrom the peers in the other partition.

All host members within the data replication framework 100 locallypersist their member ID, and they also locally persist a replicationgroup ID. Like the member ID, the group ID is a randomly assigned ID.The purpose of the group ID is to ensure that misconfigured replicationgroups cannot merge together.

For illustration, consider two replication group sets: {A, B} and {C,D}. If the first set was only seeded with A and B, then it will have noknowledge of the second set. Likewise, if the second set was only seededwith C and D, it will be unaware of the first set. The two sets aredifferent partitions. If a configuration error causes set {A, B} tobecome aware of C, the two sets union into a combined replication group:{A, B, C, D}. Since the two groups were not together initially, theirdata sets will be divergent, and a merge can create irreconcilableinconsistencies. By giving each group a unique ID, the merge is notallowed. The data replication framework 100 may log errors when such anattempt is made.

When a host first joins a replication group, it acquires the group ID ofthe first seed it was able to contact. If no seed can be contacted, thehost might be the first in the group. By default, a new host may wait upto a threshold amount of time, such as for example, 15 seconds, beforeforming its own group. Other time periods are also possible. If there isa network partition during the deployment of the new host, it mayerroneously form a new group, which causes a different kind of permanentnetwork partition. As described earlier, when a groups attempt to mergeback, the data replication framework 100 detects and logs these attemptsas errors.

If a new replication group is being formed and several hosts are allfirst, they might all form independent groups. To prevent this, a insome embodiments, a mini master election algorithm may be used duringthe join period. The new hosts detect the existence of each other asthey connect during the initial join wait period. They then exchangetheir candidate group IDs, and the one which is comparably lower thanthe other is used instead.

In other embodiments, there may be an option to disable automatic groupassignment. A manual self-generating or self-sustaining (bootstrap)process may be used in such embodiments, using two deployments, thefirst with automatic group assignment enabled, and the second withautomatic group assignment disabled.

The data replication framework 100 as described above may assume thatall members in a replication group are within the same LAN, or datacenter. Bandwidth may be expected to be high and latency may be expectedto be low. All members maintain connections to each other, and each iscapable of performing the same data replication actions. In someembodiments, the data replication framework 100 can also be used over aWAN. Replication groups across a WAN may be mostly isolated. A leaderelection process may choose which local members send changes over theWAN. The leader election process may also choose which remote membersreceive changes over the WAN. In some embodiments, there may be multipleleaders elected.

Members on each side of the WAN are provided with completely differentgroup IDs. If the same group ID were used, the two sides may learn ofeach other's existence. If this happens, they will exchange the completemembership roster and form a single fully connected network, defeatingthe formation of replication groups across the WAN.

Separate group IDs also allow for one-way replication, which may beperformed in either a LAN or a WAN framework. The separate group IDsmake it possible to aggregate several data stores into one, for example.Group ID separation also helps ensure that problems affecting one datacenter do not significantly impact services in another data center. Thereplication link can be severed in an emergency, and replication withinthe data centers continues to work normally. For LAN replication, a setof seed hosts to initially join a replication group may be configured.For WAN replication, seed hosts which reside in different groups may beconfigured. Multiple WAN destinations can be configured with thisstrategy as well.

When the core replication interface 424 first starts, it needs tocontact at least one member in the remote group. It then locallypersists its group ID associated with the configured group name. Onsubsequent starts, it does not need to immediately contact any remotegroup member, because the group ID is known. It will keep trying toconnect to at least one member, logging an error if failed, forever.Unreplicated changes will also pile up forever.

To stop replicating to a group, configuration for it must be removed.All unreplicated changes for the group will be purged when the system isrestarted with the new configuration. With WAN replication the recordersare additionally responsible for replicating changes to the remotegroup. By choosing a leader recorder/replicator, the number ofconnections established over the WAN is reduced. If multiple leadersexist at any time, extra connections are created, but this does notcorrupt the data store, since the data replication framework 100 relieson record version numbers to discard stale or duplicated changes.

In some embodiments, leader election for WAN replication may beaccomplished by examining the member IDs of all members, and to choosethe members with the lowest IDs. If one of the leaders becomesunavailable, it loses its leader status until it becomes availableagain. Leader election on the send side of replication reduces thenumber of connections over the WAN, and doing the same on the receivingside reduces it further. With no leaders at all, the number of requiredconnections scales as m*n, where m and n are the member counts on eachside. With leaders on one side, the number of connections is m or n, butwith leaders on both sides, the number of connections is 1.

The sender of a change in data may ask any member of the remote groupfor the leader. The sender only keeps a connection to the leader andsends all changes to it. In turn, the remote leader replicates allchanges to the members of its group.

When a host data store is first created or an existing data store hasbeen declared permanently unavailable, it acquires a complete copy ofthe data store from a peer in the replication group. If no peers exist,then an empty data store is created instead. During a restore, the peersupplying the copy remains online. The host data store continues toperform ordinary operations, although the host load is slightly higher.Any updates which are applied during the transfer are delayed, and theyare received after the transfer is complete.

FIG. 8 is a flow diagram illustrative of determination of querydistribution implemented by the query analysis module. One skilled inthe relevant art will appreciate that actions/steps outlined for routine800 may be implemented by one or many computing devices/components thatare associated with the query analysis module 316. Accordingly, routine800 has been logically associated as being generally performed by thequery analysis module 316, and thus the following illustrativeembodiments should not be construed as limiting. The query analysismodule 316 may be used to perform relational style queries on localhosts. The query analysis module 316 may convert the relational stylequery into formats understood by local data stores. The query analysismodule 316 may also be used for performing query analysis anddistribution, as illustrated in FIG. 8.

Query analysis starts at block 804 where queries received at the hostsare analyzed by the query analysis module 316. Although all hosts have areplicated copy of the same data store, the queries received at each ofthe hosts may have different frequencies. For example, data associatedwith a given ProductID may be queried on one host, and that ProductIDmay be a popular one, causing several queries on that host, creating ahot spot. However, a request associated with a CustomerID for that sameProductID might send the query to a different host. Therefore, thequeries are analyzed, and a determination is made of where queried datais located, at block 808. From this analysis, a distribution of requestsis inferred by the query analysis module at block 812. The distributionof requests inferred is used to try to route requests away frompotential hot spots by a router implemented by a dynamic routing requestmodule. In some embodiments, the inferred distribution may include ahistogram. The inferred distribution constantly adjusts itself, and aimsto approximate even distribution of requests across hosts. In order toachieve a more ideal distribution, a range of keys is created at block816, and the query analysis module ends at block 820.

Request routing is a method of partitioning the request stream tominimize disk accesses on the cluster of hosts. A query may include akey class, wherein a key may be any type of request, and a comparator tocompare keys. A router may determine which host every request goes to,based on dividers generated by a key store. The router may maintain anordered list of hosts, where the hosts are comparable. Such a list helpsmaintain consistency among separate applications that may be using thesame set of hosts. When a new host is added, the host is inserted inorder. The dividers are generated to account for the new host.Similarly, when a host is removed, the dividers are regenerated. The keystore stores a sorted multiset (or a set that allows for duplicates) ofthe most-recently used keys. For every n-th key coming into the router,the router also sends it to the key store. The router asks the key storeto generate a list of dividers to help the router to decide which host arequest should be routed. A divider helps the router decide to whichhost to route a request. The divider contains a key to compare therequest to, an initial host offset index, and a list of probabilities(summing to 100% or 1).

When the router is first turned on, the key store is empty, andtherefore it cannot generate dividers. Thus, the first requests may berandomly routed, and then seeded to the key store. When the key storegets full, the router begins to route by ranges.

Hosts frequently become unavailable and then available again, andsometimes more hosts need to be added or removed. When the routerdetects that there is a change in the number of available hosts, itimmediately regenerates dividers again so that the load is evenlydistributed.

The method of routing used may be based on ranges of request keys. Witha cluster of hosts, the first host would handle the first fraction ofkeys, the second host would handle the second fraction of keys, and soon. This method of routing would distribute keys to hosts evenly anddecrease the probability of disk accesses. Due to fluctuations, trendsand hotspots, the ranges should be dynamically adjusted to adapt to theconstantly changing popularity of keys. One embodiment keeps samples ofthe most recently used keys and, based off of the samples, distributesthe request stream. In some embodiments, the algorithm may employ astrategy where the range of keys that is routed to specific hostschanges based on traffic. Different hosts would have a different keyrange going to them based on what items are hot. This would help spreadthe load across the system and account for variations in load.

A parameter of the algorithm may be to determine the number of keys tostore in the keystore. A larger capacity may be better for steady,constant request streams, while a smaller capacity may be better forrequest streams that fluctuate rapidly. In some embodiments, the numberof keys may be 1000. In other embodiments, the number of keys may bemore or less than 1000.

Another parameter of the algorithm may be to determine the period ofupdating the dividers. A shorter period may make the dividers reflect amore recent past request stream but may come with a performancetrade-off. In some embodiments, the number of keys may be 1000. In otherembodiments, the number of keys may be more or less than 1000.

FIG. 9 is a flow diagram illustrative of some functions implemented bythe dynamic request routing routine module. When a request comes in, therouter searches for the insertion index of the request key in the listof dividers (which contains a key for comparison) at block 904. If theinsertion index is the size of the list (should be inserted at end oflist), the request is routed to the last host at block 912.Alternatively, if insertion index is not the same as the size of thelist, the router sends the request key to the divider for it to decideat block 908.

When the divider receives a request key, it checks if the request key isless than its own key at block 916, and if true, tells the router toroute the request to the host offset index (the first possible hostassociated to a divider) at block 924. If the request key matches thedivider's key, the divider randomly selects a host based on its list ofprobabilities, at block 920. The first element corresponds to theinitial host, the second element corresponds to the host to the right ofthe initial host, and so on.

An example is provided for illustrative purposes. Suppose there are 5hosts and a key store capacity of 20, shown below with respectivedividers:

Dividers: D — — D Indices: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 Keys:1 1 1 1 2 2 2 2 2 2 2 2 2 4 6 6 6 7 7 7 Hosts: [-------0-------][-------1-------] [-------2-------] [-------3-------] [-------4-------]

In this example, there are 2 dividers. There are no dividers at indices8 and 12 because they have the same key as a previous divider, namelythe divider at index 4. The first divider has key 2, a host offset indexof 0 (since this is the first divider, and the host to the left of thedivider has index 0), and a probability list of {0, 4/9, 4/9, 1/9}. Thefirst probability is 0, since there are no 2's in (0, 3). The next twoprobabilities are both 4/9, since there are four 2's in (4, 8) and (9,12) and nine 2's in total. Finally, the last probability is 1/9, sincethere is one 2 in (13, 16). The second divider has key 6, a host offsetindex of 3 (the host to the left of the divider is host 3), and aprobability list of {2/3, 1/3}.

If a request key of 1 comes in, the insertion index of 1 in the dividerlist is 0 (divider at index 4). The divider sees that 1 is less than itskey, so immediately returns the host offset index, 0. If a request keyof 2 comes in, the insertion index of 1 in the divider list is 0, again.The divider sees that the keys match, so it randomly selects a hostbased on the probabilities. Thus, the 2 will be never be routed to host0, it will be routed to host 1, 44.4% of the time, it will be routed tohost 2, 44.4% of the time, and it will be routed to host 3, 11.1% of thetime.

If a request key of 3 comes in, the insertion index of 3 in the dividerlist is 1 (divider at index 16). The divider sees that 3 is less thanits key, so immediately returns the host offset index, 3. If request keyof 7 comes in, the insertion index of this request key would be to theright of the dividers, so the router immediately routes it to the lasthost, host 4.

It will be appreciated by those skilled in the art and others that allof the functions described in this disclosure may be embodied insoftware executed by one or more processors of the disclosed componentsand mobile communication devices. The software may be persistentlystored in any type of non-volatile storage.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements, and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art. It willfurther be appreciated that the data and/or components described abovemay be stored on a computer-readable medium and loaded into memory ofthe computing device using a drive mechanism associated with a computerreadable storing the computer executable components such as a CD-ROM,DVD-ROM, or network interface further, the component and/or data can beincluded in a single device or distributed in any manner. Accordingly,general purpose computing devices may be configured to implement theprocesses, algorithms, and methodology of the present disclosure withthe processing and/or execution of the various data and/or componentsdescribed above.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

What is claimed is:
 1. A computing system for managing data in hostcomputing devices comprising: a plurality of hosts comprising one ormore computing devices; a sender host comprising one or more computingdevices, wherein the sender host is configured to: identify one or morerecorder hosts from the plurality of hosts, wherein the one or morerecorder hosts are configured to redundantly store changes to be sent bythe sender host to the plurality of hosts; transmit the changes to theplurality of hosts; receive a response from a first host of theplurality of hosts that the changes are received, wherein the first hostis not a recorder host; and transmit a message to the identified one ormore recorder hosts that the changes are received by at least one of theplurality of hosts, wherein the identified one or more recorder hostsredundantly store the changes to the plurality of hosts, and wherein asecond host of the plurality of hosts obtains the change from the one ormore recorder hosts while the sender host is temporarily unavailable. 2.The computing system for managing data in host computing devices ofclaim 1, wherein the identified one or more recorder hosts receive, froma requesting host, a request for data regarding the changes in thesender host and wherein the identified one or more recorder hoststransmits the data regarding the changes in the sender host to therequesting host in response to the sender host becoming temporarilyunavailable.
 3. The computing system for managing data in host computingdevices of claim 1, wherein the sender host, receives, from the at leastone of the one or more recorder hosts, data that was received from oneor more of the plurality of hosts while the sender host was temporarilyunavailable.
 4. The computing system for managing data in host computingdevices of claim 1, wherein at least one of the identified one or morerecorder hosts receives data of a change in the sender host while thesender host is temporarily unavailable, and transmits the change in thesender host to at least one of the plurality of hosts.
 5. The computingsystem for managing data in host computing devices of claim 1, whereinat least one of the identified one or more recorder hosts that receivedthe message deletes corresponding preexisting records and replaces thepreexisting records with information extracted from the message.
 6. Amethod of managing data in a set of host computing devices, the methodcomprising: receiving, by a recorder host, a designation to become therecorder host for redundantly storing changes in data stored by a firsthost; storing, by the recorder host, a pre-acknowledged entry for thefirst host; receiving, by the recorder host, a message from the firsthost that a second host received information regarding a change in thedata stored by the first host; storing, by the recorder host, theinformation regarding the change in the data stored by the first host;determining, by the recorder host, that a third host did not receive theinformation regarding the change in the data stored by the first hostand that the first host is temporarily unavailable; transmitting, by therecorder host, the information regarding the change in the data storedby the first host to the third host; and receiving, by the recorderhost, a message from the third host that the third host received theinformation regarding the change in the data stored by the first host,wherein the method is performed on a computing device comprising ahardware processor and memory.
 7. The method of managing data in a setof host computing devices of claim 6, wherein the pre-acknowledged entryis used by the first host to recover recently lost changes.
 8. Themethod of managing data in a set of host computing devices of claim 6,wherein the message includes a record version number.
 9. The method ofmanaging data in a set of host computing devices of claim 8, wherein therecord version number of the message is larger than the record versionnumber of a preexisting record.
 10. The method of managing data in a setof host computing devices of claim 8 further comprising: determiningwhether the record version number of the message is larger than therecord version number of a preexisting record; and in response todetermining that the record version number of the message is larger thanthe record version number of a preexisting record, replacing thepreexisting record with information extracted from the message.
 11. Themethod of managing data in a set of host computing devices of claim 6further comprising, in response to the first host becoming temporarilyunavailable and receiving a request from the third host for theinformation regarding the change in the data stored by the first host,transmitting, by the record host, the information regarding the changein the data stored by the first host to the third host.
 12. The methodof managing data in a set of host computing devices of claim 6 furthercomprising: receiving, by the recorder host, data from the second hostwhile the first host is temporarily unavailable; and in response to thefirst host becoming available, transmitting, by the recorder host, thedata received from the second host to the first host.
 13. The method ofmanaging data in a set of host computing devices of claim 6, furthercomprising in response to the first host becoming temporarilyunavailable, obtaining, by the recorder host, the information regardingthe change in the data stored by the first host from the second host,and transmitting the information regarding the change in the data storedby the first host to the third host.